{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 第10章 TensorFlow高层封装\n",
    "虽然原生态的TensorFlow API可以很灵活地支持不同的神经网络结构，但是其代码相对冗长，写起来比较麻烦。**为了让用户更加方便快捷地实现常用的神经网络结构，不同的组织和个人为TensorFlow提供了多种高级封装**。第6章已经简单介绍了TensorFlow的高层封装，并使用它实现了CNN，这一章将更详细地介绍几种最常用的TensorFlow高层封装。\n",
    "\n",
    "## 10.1 TensorFlow高层封装总览\n",
    "**目前比较主流的TensorFlow高层封装主要有四个，分别是TensorFlow-SLim、TFLearn、Keras和Estimator。**\n",
    "\n",
    "**1. TensorFLow-Slim**\n",
    "\n",
    "TensorFLow-Slim是Google官方给出的相对较早的TensorFlow高层封装，Google通过TensorFLow-Slim开源了一些已经训练好的图像分析的模型，所以目前在图像识别问题中TensorFLow-Slim仍被较多地使用，下面给出一个简单的样例，介绍了如何使用TensorFLow-Slim在MNIST数据集上实现LeNet5模型："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From <ipython-input-1-8f1656bb275a>:68: read_data_sets (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use alternatives such as official/mnist/dataset.py from tensorflow/models.\n",
      "WARNING:tensorflow:From d:\\python3\\tfgpu\\dl+\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\datasets\\mnist.py:260: maybe_download (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please write your own downloading logic.\n",
      "WARNING:tensorflow:From d:\\python3\\tfgpu\\dl+\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\datasets\\mnist.py:262: extract_images (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use tf.data to implement this functionality.\n",
      "Extracting ../../datasets/MNIST_data\\train-images-idx3-ubyte.gz\n",
      "WARNING:tensorflow:From d:\\python3\\tfgpu\\dl+\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\datasets\\mnist.py:267: extract_labels (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use tf.data to implement this functionality.\n",
      "Extracting ../../datasets/MNIST_data\\train-labels-idx1-ubyte.gz\n",
      "WARNING:tensorflow:From d:\\python3\\tfgpu\\dl+\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\datasets\\mnist.py:110: dense_to_one_hot (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use tf.one_hot on tensors.\n",
      "Extracting ../../datasets/MNIST_data\\t10k-images-idx3-ubyte.gz\n",
      "Extracting ../../datasets/MNIST_data\\t10k-labels-idx1-ubyte.gz\n",
      "WARNING:tensorflow:From d:\\python3\\tfgpu\\dl+\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\datasets\\mnist.py:290: DataSet.__init__ (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use alternatives such as official/mnist/dataset.py from tensorflow/models.\n",
      "After 0 training step(s), loss on training batch is 2.30333, accuracy on validation is 0.065600.\n",
      "After 1000 training step(s), loss on training batch is 0.426152, accuracy on validation is 0.851800.\n",
      "After 2000 training step(s), loss on training batch is 0.145496, accuracy on validation is 0.967800.\n",
      "After 3000 training step(s), loss on training batch is 0.0508656, accuracy on validation is 0.976000.\n",
      "After 4000 training step(s), loss on training batch is 0.0511929, accuracy on validation is 0.980800.\n",
      "After 4999 training step(s), loss on training batch is 0.0931923, accuracy on validation is 0.982400.\n",
      "Final test accuracy is 0.981000.\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "import tensorflow.contrib.slim as slim\n",
    "import numpy as np\n",
    "\n",
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "\n",
    "\n",
    "# 通过TensorFlow-Slim来定义LeNet-5的网络结构。\n",
    "def lenet5(inputs):\n",
    "    # 将输入转化为一个4维数组，第一维是batch大小，后面三维表示一张图片\n",
    "    inputs = tf.reshape(inputs, [-1, 28, 28, 1])\n",
    "    \n",
    "    # 定义第一卷积层。可以看出，TensorFLow-Slim定义的网络结构并不需要用户去关心如何声明和初始化变量，\n",
    "    # 而只需要定义网络结构即可。\n",
    "    net = slim.conv2d(inputs, 32, [5, 5], padding='SAME', scope='layer1-conv')\n",
    "    net = slim.max_pool2d(net, 2, stride=2, scope='layer2-max-pool')\n",
    "    \n",
    "    # 定义第二卷积层\n",
    "    net = slim.conv2d(net, 64, [5, 5], padding='SAME', scope='layer3-conv')\n",
    "    net = slim.max_pool2d(net, 2, stride=2, scope='layer4-max-pool')\n",
    "    \n",
    "    # 直接使用TensorFLow-Slim封装好的flatten函数将4维矩阵将为2维，用户不需计算通过卷积后的矩阵大小。\n",
    "    net = slim.flatten(net, scope='flatten')\n",
    "    \n",
    "    # 通过TensorFLow-Slim定义全连接层，500是节点数。\n",
    "    net = slim.fully_connected(net, 500, scope='layer5')\n",
    "    net = slim.fully_connected(net, 10, scope='output')\n",
    "    \n",
    "    return net\n",
    "\n",
    "\n",
    "# 通过TensorFLow-Slim定义网络结构，并使用之前章节中给出的方式训练定义好的模型。\n",
    "def train(mnist):\n",
    "    # 1. 定义网络输入输出\n",
    "    x = tf.placeholder(tf.float32, [None, 784], name='x-input')\n",
    "    y_ = tf.placeholder(tf.float32, [None, 10], name='y-input')\n",
    "    \n",
    "    # 2. 定义前向传播、损失函数、反向传播\n",
    "    y = lenet5(x)\n",
    "\n",
    "    cross_entropy = tf.nn.sparse_softmax_cross_entropy_with_logits(logits=y, labels=tf.argmax(y_, 1))\n",
    "    loss = tf.reduce_mean(cross_entropy)\n",
    "\n",
    "    train_op = tf.train.GradientDescentOptimizer(0.01).minimize(loss)\n",
    "    \n",
    "    # 其他准备，feed、accuracy\n",
    "    feed_valid = {x:mnist.validation.images, y_:mnist.validation.labels}\n",
    "    feed_test = {x:mnist.test.images, y_:mnist.test.labels}\n",
    "    correct_pred = tf.equal(tf.argmax(y, 1), tf.argmax(y_, 1))\n",
    "    accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))\n",
    "    \n",
    "    # 3. 建立会话，训练\n",
    "    with tf.Session() as sess:\n",
    "        tf.global_variables_initializer().run()\n",
    "        for i in range(5000):\n",
    "            xs, ys = mnist.train.next_batch(100)\n",
    "            _, loss_value = sess.run([train_op, loss], feed_dict={x: xs, y_: ys})\n",
    "\n",
    "            if i % 1000 == 0 or i == mnist.validation.images.shape[0] - 1:\n",
    "                acc_valid = sess.run(accuracy, feed_dict=feed_valid)\n",
    "                print(\"After %d training step(s), loss on training batch is %g, accuracy on validation is %f.\" % (i, loss_value, acc_valid))\n",
    "                \n",
    "        acc_test = sess.run(accuracy, feed_dict=feed_test)\n",
    "        print('Final test accuracy is %f.' % acc_test)\n",
    "                \n",
    "                \n",
    "def main(argv=None):\n",
    "    mnist = input_data.read_data_sets(\"../../datasets/MNIST_data\", one_hot=True)\n",
    "    train(mnist)\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    main()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "从以上代码可以看出，**TensorFlow-Slim主要的作用是使模型定义更加简洁，基本上每一层网络可以通过一句话来实现。除了对单层网络结构，TensorFlow-Slim 还对数据预处理、损失函数、学习过程、测试过程等都提供了高层封装**不过因为TensorFlow-Slim的这些封装使用得并不广泛，所以本书不做详细介绍，感兴趣的读者可以参考GitHub上TensorFlow-Slim的[代码库](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/slim)。\n",
    "\n",
    "**TensorFlow-Slim最特别的一个地方是它对一些标准的神经网络模型进行了封装，比如VGG、Inception以及ResNet**，而且Google开源的训练好的图像分类模型基本都是通过TensorFlow-Slim实现的。第6章介绍迁移学习时已经使用过通过TensorFlow-Slim定义的Inception-v3模型了。更加详细的、通过TensorFlow-Slim开源的训练好的模型列表可以在GitHub上[找到](https://github.com/tensorflow/models/tree/master/research/slim/nets)。\n",
    "\n",
    "**2. TFLearn**\n",
    "\n",
    "与TensorFlow-Slim相比，TFLearn是一个更加简洁的TensorFlow高层封装。通过TFLearn可以更加容易地完成模型定义、模型训练以及模型评测地全过程。TFLearn没有集成在TensorFlow地安装包中，故需要单独安装：`pip install tflearn`。以下代码展示了如何使用TFLearn在MNIST数据集上实现LeNet5模型："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Step: 17199  | total loss: \u001b[1m\u001b[32m0.02664\u001b[0m\u001b[0m | time: 7.026s\n",
      "| SGD | epoch: 020 | loss: 0.02664 - acc: 0.9917 -- iter: 54976/55000\n",
      "Training Step: 17200  | total loss: \u001b[1m\u001b[32m0.02501\u001b[0m\u001b[0m | time: 8.214s\n",
      "| SGD | epoch: 020 | loss: 0.02501 - acc: 0.9925 | val_loss: 0.03196 - val_acc: 0.9887 -- iter: 55000/55000\n",
      "--\n"
     ]
    }
   ],
   "source": [
    "import tflearn\n",
    "from tflearn.layers.core import input_data, fully_connected\n",
    "from tflearn.layers.conv import conv_2d, max_pool_2d\n",
    "from tflearn.layers.estimator import regression\n",
    "\n",
    "import tflearn.datasets.mnist as mnist\n",
    "\n",
    "# 读取mnist数据集\n",
    "trainX, trainY, testX, testY = mnist.load_data(\"../../datasets/MNIST_data\", one_hot=True)\n",
    "\n",
    "# 将图像数据reshape成CNN输入的格式\n",
    "trainX = trainX.reshape([-1, 28, 28, 1])\n",
    "testX = testX.reshape([-1, 28, 28, 1])\n",
    "\n",
    "# 构建神经网络，这个过程和tensorFlow-Slim比较类似\n",
    "# input_data定义了一个placeholder来接入输入数据\n",
    "net = input_data(shape=[None, 28, 28, 1], name='input')\n",
    "# 通过TFLearn封装好的API定义一个深度为5，过滤器为5*5，激活函数为ReLU的卷积层\n",
    "net = conv_2d(net, nb_filter=32, filter_size=5, activation='relu')\n",
    "# 定义了一个过滤器为2*2的最大池化层\n",
    "net = max_pool_2d(net, kernel_size=2)\n",
    "# 类似地定义其他地网络结构\n",
    "net = conv_2d(net, 64, 5, activation='relu')\n",
    "net = max_pool_2d(net, 2)\n",
    "net = fully_connected(net, 500, activation='relu')\n",
    "net = fully_connected(net, 10, activation='softmax')\n",
    "\n",
    "# 使用TFLearn封装好的函数定义学习任务。指定优化器为sgd，学习率为0.01，损失函数为交叉熵\n",
    "net = regression(net, optimizer='sgd', learning_rate=0.01, \\\n",
    "                 loss='categorical_crossentropy')\n",
    "\n",
    "# 通过定义地网络结构训练模型，并在指定的验证集上验证模型效果。\n",
    "# TFLearn将模型的训练过程封装到一个类中，这样可以减少非常多的冗余代码\n",
    "model = tflearn.DNN(net, tensorboard_verbose=0)\n",
    "\n",
    "model.fit(trainX, trainY, n_epoch=20, validation_set=([testX, testY]), show_metric=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "从以上代码可以看出，**使用TFLearn训练神经网络的流程也是一样的：先定义神经网络的结构，再使用训练数据来训练、模型**。\n",
    "\n",
    "**TFLearn vs. 原生态TensorFlow：**\n",
    "\n",
    "- TFLearn不仅使神经网络结构定义更加简洁，还将模型训练的过程也进行了封装。另外，在定义神经网络的前向传播过程之后， TFLearn可以**通过regression函数来指定损失函数和优化方法**。更方便的是，**不仅TFLearn能很好地封装模型定义，tflearn.DNN也能很好地封装模型训练的过程**。通过fit函数可以指定训练中使用的数据和训练的轮数。这样避免了大量的冗余代码。\n",
    "\n",
    "**TFLearn vs. Keras&Esitimator:**\n",
    "- 相同点：封装的方式上基本一致，主要也是针对模型定义和模型训练两个部分；\n",
    "- 不同点：Keras和Estimator都己经加入了TensorFlow代码库，而且它们是使用最为广泛的TensorFlow高层封装。\n",
    "\n",
    "限于篇幅，本书不在介绍TFLearn的复杂方法的使用，感兴趣的读者可以参考[TFLearn官网](http://tflearn.org/)上的相关内容。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
